Inter - Observer Reliability

Just as scientific records can be vulnerable to variation between different pieces of recording equipment, so observational records are subject to differences in how observers score behaviour. However one can judge how much confidence to place in one's results by assessing the reliability of the records of one's variables. In this paper we shall consider only inter-observer reliability or scorer reliability (ANASTASI, i968) of direct observation of discrete events, however it should be remembered that observer differences are not the only source of error in observational studies. Indeed the amount of variation that is treated as error will depend on the aim of the study. For example, if an experimenter assumes that repeated observational measures within a short time interval should be consistent, he needs to assess both the error due to variation in subjects over time as well as the error due to observer differences. Inter-observer reliability is an estimate of the latter error only, although the methods of its calculation outlined in this paper are applicable to other types of reliability (see ANASTASI, i1968, for further discussion). Inter-observer reliability is often reported simply as a figure or a percentage with little or no explanation of how the figure was derived. For example, in a recent volume of a well known journal concerned with studies of children, out of the I4 papers using observational data, only six reported clearly how they had derived the reliability figures, and these six included at least three different methods; three gave only a bare figure which could have been derived in a number of different ways; the remaining five papers gave a description of their methods that was in some way ambiguous, or quoted